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ABSTRACT 

Developed for educators and trainers in the U.S. 
Navy^ this Instructional Quality Inventory (IQI) is designed to 
provide quality control and evaluation procedures for instructional 
development. A by-step instructional development approach is provided 
for each of five categories: objective adequacy^ test consisten cy ^ 
test adequacy^ presentation consistency ^ and presentation adequacy. 
Amonq other things^ objective adequacy^ test consistency^ and 
presentation consistency are examined according to content type: 
face, concept^ procedure^ rule^ or principle. The inventory concludes 
with checklists for each of the five inve^ntory categories. For 
example, for objective adequacy the checklist inquires about the 
objective's correctness^ classif ica t ion^ and appropriateness. 
(CSS) 
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FOREWORD 



The Instructional Quality Inventory (IQI) was developed in support of 
Navy Decision Coordinating Paper, Education and Training Development 
(NDCP-Z0108-PNh under subproject P.30A, Adaptive Experimental Approach 
to Instructional Design, and the sponsorship of the Director of Naval 
Education and Training (OP-99). The overall objective of the subproject 
is to develop an empirically-based instructional design support system 
to aid developers in deciding on instructional alternatives based on 
cost/benefits and specified resource limitations. The purpose of the 
IQI, which was originally called the Instructional Strategy Diagnostic 
Profile (ISDP), is to provide „qual ity control and/or evaluation procedures 
for instructional development. 

A number of reports have been published on the IQI/ISDP. The first pro- 
vided an interim training manual for the ISDP (NPRDC Special Report 77-14) , 
and the next three addressed its empirical and workshop evaluations (NPRDC 
Technical Reports 77-25 and 77-43 and Special Report 78-17). As a result 
of these evaluations, the ISDP was extensively revised and retitled as 
the IQI. These revisions were included in NPRDC Technical Note 78-5, 
which provided an interim training manual for the IQI. 

This report is the fourth in a series of four on IQI procedures. It provides 
a Job Performance Aid for the IQI process, including a list of all the IQI 
procedures. The other three reports are: 

1. The Instructional Quality Inventory: Volume I, Introduction and 
Overview (NPRDC Special Report 79-3). 

2. An IQI User's Manual, which will include a complete description of 

all IQI procedures, and provide examples of their use (to be published 
in early 1979). 

3. An IQI Training Workbook, which will provide additional examples, and 
practice on the IQI procedures (to be published in early 1979). 

When these four reports have been publisned, previous training manuals (NPRDC 
Special Report 77-14 and Technical Note 78-5) will have been superseded. 

The IQI is intended for use by the Chief of Naval Education and Training; 
the Chief of Naval Technical. Training; the Chief of Naval Education and- 
Training Support (specifically, the Instructional Program Development Centers); 
the Commander Training Command, Atlantic; the Commander Training Command, 
Pacific; and all other Navy activities concerned with the development, revision, 
or acquisition of instructional ^programs. Prospective users of the IQI are 
invited to contact this command fo. assistance in implementation. 



DONALD F. PARKER 
Commanding Officer 
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OBJECTIVE ADEQUACY 

STEP 1: ENTER the COURSE TITLE and OBJECTIVE NUMBER at the top of the form. 

STEP 2: Determine whether or not the OBJECTIVE is CORRECTLY STATEDl,,,,-*——** 

2a: Are the CONDITIONS under which student performance is expected specified? 

ENVIRONMENT: PHYSICAL (weather, time of day. lighting, etc) 
* SDCIAL (isolation, individu^*!. tF.am. audience, etc.) 

PSYCHOLOGICAL (fatigue, strt relaxed, etc.) 

INFORMATION: GIVEN INFORMATION (scenario, fnrmula. values, etc) 
CUES (signals for starting 'c^ stopping) 
. SPECIAL INSTRUCTIONS 

RESOURCES: JOB AIDS (cards, charts, graphs, checklists, etc) 
EQUIPMENT. JOOLS 
TECHNICAL MANUALS 

.2b: Are the STANDARDS which the student performance must meet specified? 

PERFORMANCE: COMPLETENESS (how much of the task must be performed) 
ACCURACY (how well must each task be performed) 
TIME LIMIT (how much time is allowed) 
RATE (how fast must task be done) 

PRODUCT: ■ COMPLETENESS (what must finished product contain) 
QUALITY (what objective standard must product meet) 
JUDGEMENT (what subjective opinions must product satisfy) 

< 2c: Is the ACTION the student must perform specified? 

Is an action verb used to specify what the student must do? 

Is only one action stated in the objective? 

STEP 3: Determine whether or not the OBJECTIVE is CLASSIFIABLE? Does the OBJECTIVE fit 
in one and only one cell of the table below? 
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CARD 1 



FACT 

RECALL OR 

RECOGNIZE 

NAMES> 

PARTS> 

DATES> 

PLACES^ ETC, 



CONCEPT 

REMEMBER 
CHARACTER IS- 
TICS> OR 
CLASSIFY OB-^ 
JECTS^ EVENTS 
OR IDEAS AC- 
CORDING TO 
CHARACTERISTICS 



PROCEDURE 

SEQUENCE OF 
STEPS REMEM- 
BERED OR USED 
IN A SINGLE 
SITUATION OR 
ON A SINGLE 
PIECE OF 
EQUIPMENT 



RULE 

REMEMBER OR 
USE A SEQUENCE 
OF STEPS WHICH 
APPLY ACROSS 
SITUATIONS 
OR ACROSS 
EQUIPMENTS 



PRINCIPLE 

REMEMBER^ OR 
INTERPRET OR 
PREDICT, WHY 
OR HOW THINGS 
HAPPEN, OR 
CAUSE-EFFECT 
RELATIONSHIPS 



REMEMBER - recall or 

RECOGNIZE FACTS, CON- 
CEPT DEFINITIONS, STEPS 
OF PROCEDURES OR RULES, 
STATEMENTS OF PRINCIPLE 










9 


USE- UNAIDED - tasks which require 

CLASSIFYING, PERFORMING A PROCEDURE,' 
USING A RULE, EXPLAINING OR PREDICTING 
WITH NO AIDS EXCEPT MEMORY. 










USE-AIDED - SAME AS use-unaided, 

EXCEPT JOB AIDS ARE AVAILABLE. 










. (Continued on other side) 
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STEP 4: Determine whether or not the OBJECTIVE is APPROPRIATE? 

4a: Are the CONDITIONS appropriate for the work to be performed on the job or for 



later training? 

4b: Are the STANDARDS appropriate for the work to oe performed on the job or for 
later training? 

4c: Is the TASK LEVEL of the ACTION appropriate for the work to be performed on the 
job or for later training? 

4d: Is the CONTENT TYPE of the ACTION appropriate for the work to be performed on the 
job or for later training? 
) 

4e: If this*^ objective is REMEMBER, is there a later USE objective? 

4f: If this objective is USE-UNAIOEO, is there a previous REMEMBER objective? 

4g: If this objective is USE-AIOEO, is the aid adequate, or are other objectives 
on the aid included? 

Notey if the answer to 4d, 4ey or 4g is yeSj and if thp associated objective is 
be taught in the present course, evaluate that objective next and keep the vela 
together throughout the IQI evaluation. 





TEST CONSISTENCY 
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STEP 1: ENTER the COURSE TITLE and OBJECTIVE NUMBER at the top of the form. 

STEP 2: ENTER the TEST ITEM NUMBERS for thY items associated with this objective o.. the form. 

STEP 3: DETERMINE whether or not the CONDITIONS in each item, or the CONDITIONS urder which the 
items are administered, match the conditions in the objective. 

STEP 4: DETERMINE whether o\ not the STANDARDS in each item, or the STANDARDS frr scoring each * 
item, match the standards in the objective. 

STEP 5: DE TERMINE whether or not the ACTIONS in each item match the action of t.ifj. objective. 

STEP 5a: Determine the TASK LEVEL and CONTENT TYPE cf each test ite^i. 

STEP 5b: Determine whether these match the TASK LEVEL and CONTENT TYPE of the objective. 

STEP 6: DETERMINE whether or not each item is typical of the job to be performed after training, 
or is a necessary qualification for later training. 

STEP 7: DETERMINE whether or not the FORMAT of each item is APPROPRIATE for the TASK LEVEL and 
CONTENT TYPE. Use the table below: 



fONTENT TYPE 



TASK LEVEL FACT CONCEPT PROCEDURE RULE PRINCIPLE 



REMEMBER 


for RECOGNITION: 

matching 

true-fal se 

multiple choice 
for RECALL: . 

Short answer 

fill-in 

1 i sting 


short answer 
fill-in 
1 i sting 


short answer 

fill-in 

listing 


short answer 
fill-in 
1 i sting 


short answer 
fill-in 
1 is ting 


USE-UNAIDED 




performance 
ma tching 
true-false 
multiple choice 
short answer 
fill-in 


performance 
true-false 
multiple choice 
short answer 
fill-in 


performance 
true-false 
multiple choice 
short answer 
fill-in 


performance 
true-false 
multiple choice 
short answer 
fill-in 


USE-AIDED 




performance 
matching 
true-false 
multiple choice 
short answer 
filT-in 


performance 
true-fal se 
multiple choice 
short answer 
fill-in 


performance 
true-false 
maltiple choice 
short answer 
fill-in 


performance 
true-false 
multiple choice 
short answer 
fill-in 
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CARD 3 



STEP 4: 

STEP 5; 

STEP 6: 



MULTIPLE CHOICE items: 



MATCHING items: 



STEP 1: DETERMINE whether or not each item is CLEAR. Inetruciione for completing the item must specify 
■ what vesponse the student is expected to make, 

STEP 2: DETERMINE whether or not each item is UNAMBIGUOUS. Each item must have one and only one correct 

response, and the item must be interpretable in only one way. That is, items must not he confusing, 

STEP 3: DETERMINE whether or not each item is WELL CONSTRUCTED. Different criteria apply to different 
item formats: 

TRUE-FALSE items: An item should include only one statement to be 

judged true or false. 
Negative statements should he avoided. 
Don't use words like "never," "always," etc. 
Item statements should be short. 

All alternatives should be plausible , 
Negatives in the item stem should be highlighted, 
Repetitive phrases should be placed in the stem, 

not in^.the alternatives, 
Altemattves like "all of the above," "A and B only," 
should be avoided. 

Instructions should explain the contents of each column, 

and explain the basis for matching. 
Instructions should specify how many times each ayiswer 
, may be used. 

The choice column should include exti^a answers unless 
answen^s may be used more than once. 

The blank should be at or near the end of the sentence. 
One and only one- phrase should correctly complete the item, 
hhdtiple blanks should be avoided. 
Blanks should require key words. 

The required answer should be kept short. 
The directions to the student should specify how the 

item will be scored. 
The scoy*ing key should identify allowable synonyms or 
. ' altei*natives. 

The directions should specify the number of things to 
be' listed (if appropriate for the objective, and if 
the number of thinnp ia not a hint), \ 
The directions should specify whether or not order is 
important. If so, the scoring key should score ^, 
sequence separately, ' 
The scoring key should identify allowable synonyms or 
alterrxitives, and should specify different weights 
if appropriate. 

The directions should clearly explain what the student 

is to do and how the item will be scored. 
The scoring key must specify all criteria the performance 
must meet, such as completeness, accuracy, quality, 
time limit, rate, etc. If steps in the performance 
are scored, a checklist should be provided, 

DETERMINE whether or not each .item is FREE of HINTS. An item should not give away' the answer 
to itself or to any other item on the test. The granmar of multiple-choice and fill-in items 
should not give hints to answers, 

DETERMINE whether or not the items allow for COMMON ERRORS to be made. 

For USE-level objectives, DETERMINE whether or not there are ENOUGH ITEMS to test the 
objective adequately, and to reflect the range of performance required on the job. 



FILL-IN items: 



SHORT ANSWER items: 



LISTING items: 



PERFORMAN€^ items 
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STEP 1: 
STEP 2: 
STEP 3: 




PRESENTATION CONSISTENCY 

ENTER the COURSE TITLE and OBJECTIVE NUMBER at the top of the form. 

LOCATE the section of the presentation related to this objective. 

OETERMINE if the reouired PRESENTATION COMPONENTS are present for the task level 
of the objective. (Use the table below). 
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TASK LEVEL 


STATLMENT 


PRACTICE 
REMEMBERING • 


EXAMPLES 


PRACTICE 
USING 


REMEMBER 


required 


j'cquired 


not 

requ i red 


not 

required 


USE-UNAIOEO 


required un 
associated 
objective w 
re a en 1 1 :^ 


less the 
REMEMBE.R 
3S tauaht 


required 


required 


USE-AIDEO 


replace J 
atari'mfint 


}lO t 


required 
u^lth ai.i 


required 
loith aid 



STEP 4: OETERMINE whether or not each required PRESENTATION COMPONENT is COMPLETE for the content 
type of the objective (Use the table below). Examples and practice items must also match 
the task level of the objective. ' 



PRESENTATION 
COMPONENT 



CONTENT TYPE OF THE OBJECTIVE 





FACT 


CONCEPT 


PROCEOURE 


RULE 


PRINCIPLE 


STATEMENT 


complete fact 


all critical 


all steps are 


all steps and 


all causes. 




presented 


characteristics 


given in the 


branching 


effects, and 






and their 


correct order 


decisions are 


relationships 






combination 




given in the 


are given 






are given 




correct order 




PRACTICE 


recall or recog- 


recall of 


recall of all 


recall of all 


recall of all 


REMEMBERING 


nition required 


concept def- 


steps in correct 


steps and branch 


causes, effects. 






inition required 


order required 


decisions, in 


relationships 










correct order 


required 










required 






For all content 


types: Practice Remembering items must 


be the same as the 


test item. 






They must be the same format as 


the test item. 








All practice items must include 


feedback. 




EXAMPLES 


not 


examples show 


application of 


application of 


interpretation 




applicable 


all critical 


the procedure 


each step or 


or prediction 






characteristics 


must be shown 


branching 


based on causes. 






required for 


and steps must 


decision must 


effects, and 






classification. 


be shown in the 


be shown in the 


relationships 






non-examples show 


correct order 


correct order 


must be shown 






absence of 












critical 












characteristics 








PRACTICE 


not 


classification 


all steps must 


all steps and 


explanation or 


USING 


applicable 


of both examples 


be performed 


branching 


prediction 






and non-examples 


in the correct 


decisions must 


based on the 






is required 


order 


be performed in 


principle is 










the correct order 


required 



For all content types: 
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For CONCEPTS, RULES, 
and PRINCIPLES: 



Practice Using items must reflect what is to be done on the 

job or in l^i^r training. 
The task/contei.t Jevel , conditions, and standards must match 

the test item and objective. 
The practice item format must be the same as the test item format. 
All practice items must include feedback. , \ 

Sofne practice i jems should be different than either the test 
items or the examples. (Common error items might be the same.) 

3 



STEP 1: 



PRESENTATION ADEQUACY 



DETERMINE whether or not each required PRESENTATION COMPONENT meets the general 
ADEQUACY criteria below. 




Instructional 
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CARD 5 



SEPARATED: 



IDENTIFIED: 



Statements, Examples, o" Practice must be SEPARATED from the 
rest of the instruction. There are different ways componentB 
may he separated: <• 

a. Set off the component with a box, 

h. Use a different color or type face, or underline. 

c. Place on a separate page, or in a special place on the page. 

d. For audio, movies, or lectures, pause before introducing the 
component. 

Statements, Examples, and Practice must be IDENTIFIED so the 
student knows what they are, and can locate them. Labels can 
be used to identify different components: 



Defin^'-tion cf 
Proceaure for 
Key Point: 



Example 
Demonstration 



Practice 
Test .Yourself 



CLEARLY STATED: 



Statements, Examples, and Practice must be CLEARLY STATED so the 
student can understand them. The followinn criteria should he 
used: 



a. The READING LEVEL must be appropriate for the students, 
' b. The presentation should not be confusing, vague, or too wordy. 

c. All essential information should be present; the student 
should not be referred to other places to obtain information. 

d. All presentations should be PEF^ORmNCE-ORIENTED, not 
topic-oriented. 

STEP 2: DETERMINE whether or not each required PRESENTATION COMPDNEN' -ets the specific ADEQUACY 



criteria ^eiow: 



STATEMENTS: 



HELP: 



CLEARLY STATED: 



In addition to the statement, the instruction should. include something 
to help the student better understand and remember the statement. 

Methods of providing help include: 

a. Giving a MNEMONIC (memory trick) or other memory aid. 

b. Representing the statement with pictures, symbols, flowcharts, etc. 

c. Explaining how the statement relates to something the student 
already knows, how it fits in the course, wny it is 'important. - 

d. Giving more explanation about what the . statement means, 

Iji addition to the criteria above, statements for different content types 
must meet the criteria belowT 

CONCEPTS: Give a "decision rule or search strategy for classification. 
PROCEDURES: Each step should have only- one action. ' • 

RULES: If a formula is used, symbols must be defined. 



(Continued on other side) 
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EXAMPLES: HELP: Examples should include additional ihTormation to help the student 

understand. Some types of HELP for different CONTENT TYPES are given below: 



1 ' HELP for CONCEPTS: HighligHt cr' tica'Lcharacterietioe, 

** ' • , Explain the reaeone for claeeification. 

J Illustrate the uee of the decision rule or 

• * ' ^. • 'search strategy. 

HELP for PROCEDURES or RULES: Explain the steps. 

Give inore information on hoi) to perform 
each step. 

' Give h. demonstration of each step, 

>. Give ftowahcA*tSj tables^ etc. 

- , HELP for PRI^JCIPLES: Highlight important causes j effects j relationships. 

Give additional information about how the prinr*'^fe^ 
applieSy or why it doesn't. 

MATCHING:' Examples should be matched with non-e^iamples according to the presence 

(concepts only) and absence of each critical characteristic^. 

EASY to HARD: . Early examples s^ould^be simplified. Conditions and standards may be 
relaxed on early example so the student can learn gradually. 

ENOUGH EXAMPLES: There should be enough examples to cover the content area thoroughly. 

. ' <> 

For CONCEPJi: A\t least one example-nonexample pair for each 

.critical characteristic. 
For PRDCEDG^ES or RULES: One example for each important step. 

examples for all possible branches or decisions. 

COMMON ERRORS: Examples should ^show why coirmon errors are wrong. 

PRACTICE FREE OF HINTS: The-practice items should not include anything that won't appear on 

REMEMBERING the test, or on the job. 

or . ' ' 

PRACTICE EASY to HARD: (USING onl") Early practice items may be simplified, or conditions and 

USING: ' standards may be relaxed*. Final practice items should be typical of the 

job. ■» • 

COWON ERRORS: * (USING only) Practice should include the opportunity for comnon errors, 
so they can be corrected. 

ENOUGH PRACTICE: There should be'enough practice $o the student has an adequate opportunity 
^ to learn. 

, For CONCEPTS: Practice for presence or absence of all critical 

' characteristics. . 

For PROCEDURES or- RULES: Prcctice for all possible branches or \ 
; decjs^ions. ' ' 

FEEDBACK: The FEEQBACK must be SEPARATED and IDENTIFIED . , . 

V 

FEEDBACK HELP: The FEEDBACK should include HELP l;ke the help fOr^ examples. 
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COURSE TITLE 



OBJECTIVE n 



OBJECTIVE ADEQUACY 



OBJECTIVE CORRECTLY STATED? 



REVISIONS 

CONDITIONS y n 



STANDARDS y ■ n 
ACTION . y n 



OBJECTIVE CLASSIFIABLE? (iN ONE AND ONLY ONE WAY) y » 

. (oivale the alassifiaation) REVISIONS 

FACT 

REMEMBER 

CONCEPT 

USE-UNAIDED 

PROCEDURE 

USE-AIDED 

RULE 

PRINCIPLE 



OBJECTIVE APPROPRIATE? (POR JOB OR LATER TRAINING) 

REVISIONS 

CONDITIONS APPROPRIATE? y n - 

STANDARDS APPROPRIATE? ' y n 

■ ACTION - TASK LEVEL APPROPRIATE? y n. 

- CONTENT "TYPE APPROPRIATE? y n 

if REMEMBER, is there a later 

USE objective? y n 

if USE-UNAIDED, is there a previous 

REMEMBER objective? ' y n 

if USE-AIDED, is the aid adequate, 

or are other pbjectives on the 

aid included? , y ^ 



COURSE TITLE 



UBJHCTl W 



TEST CONSISTENCY and ADEQUACY 



CONSISTENCY 



teat test 
item item 
if revisions # 



r'ev%s%ons 



test 
item 

# revision 



CONDITIONS SAME (as objective)? 

STANDARDS ^AME? 

ACTION (TASK/CONTENT) SAME? 

TYPICAL OF JOB? 

FOP.'IAT APPROPRIATE? 



ADEQUACY 

T ^ 



: CLEAR? 

UNAMBIGUOUS? 

WELL-CONSTRUCTED? 
^FREE of HINTS? 



Are COMMON ERRORS tested? 
Are there ENOUGH ITEMS? 



y n 

y n 

y n 

y n 

y n 



y n 

y n 

y n 

y n 



y n 
]( n 



y ^ 

y n 

y n 

y « 

y n 



y n 

y n 

y n 

y n 



REVISIONS 



y n 

y n 

y n 

y n 

y n 



y n 

y n 

y n 

y n 
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COURSE TITLE 



OBJECTIVE if 



PRESENTATION CONSISTENCY and ADEQUACY 



CONSISTENCY 

REVISIONS 

Required PRESENTATION 
COMPONENTS Present? ^ 



Required PRESENTATION 
COMPONENTS Complete? 



ADEQUACY 



STATEMENT ADEQUATE? 

Separate? 
Identified? 
Clear? 
Hel p? 



EXAMPLES ADEQUATE? 



Separate? 


y 


n 


Identified? 


y 


n 


Clear? 


y 


n 


Help? 


y 


n 


Matching (concepts only)? 


y 


n 


Easy to hard? 


y 


n 


Enough? 


y 


n 


Common errors? 


y 


n 



PRACTICE ADEQUATE: 



Separate? 


y 


n 


Identified? 


y 


n 


Clear? 


y 


n 


Free of hints? 


y 


n 


Easy to hard? 


y 


n 


Common errors? 


u 


n 


Enough? 


y 


n 


Feedback Separate? 


y 


n 


Feedback Identified? 


y 


n 


Feedback Help? 


y 


n 



ErJCa ''eedback Help? u " 
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